Skip to content

refactor(tests): consolidate test tooling under tests/tools - #499

Merged
16bit-ykiko merged 1 commit into
mainfrom
refactor/test-tools
Jul 10, 2026
Merged

refactor(tests): consolidate test tooling under tests/tools#499
16bit-ykiko merged 1 commit into
mainfrom
refactor/test-tools

Conversation

@16bit-ykiko

@16bit-ykiko 16bit-ykiko commented Jul 10, 2026

Copy link
Copy Markdown
Member

Summary

All Python test framework/runner code moves out of the suite trees into one flat tests/tools/ package (net −32 lines despite two new modules), with duplication cleaned up along the way.

Layout

tests/tools/
  compile_commands.py   # every compile_commands.json writer: cmake generation,
                        # static test data, per-test entries (stdlib-only)
  client.py             # CliceClient
  lifecycle.py          # server spawn / graceful shutdown / clean-exit gate / ports
  checks.py             # observation & assertions: diagnostics + anomalies + waits
  workspace.py          # on-disk sources, document edits, cache inspection
  injection.py          # protocol fuzz builder
  replay.py             # smoke-test runner
  prepare.py            # editor E2E fixture prep
  stress.py             # manual stress tool
  • tests/tools/ top level stays stdlib-only importable: the editor pixi env (no pygls) runs prepare.py, which imports compile_commandstests/tools/__init__.py is deliberately empty.
  • conftest.py shrinks to fixtures and hooks (312 → 165 lines); lifecycle helpers move to tests/tools/lifecycle.py where tests import them directly.
  • The cache_dir / worker-count test policy now lives once, in CliceClient.initialize — agentic tests that initialize directly get the same 3-process default as fixture-based ones, and their redundant per-test init_options are gone. stress.py explicitly opts back into server-side worker autoscaling (it exists to stress real pools).
  • Resolves the tests/stress.py vs tests/integration/stress/ name clash; decorative section-separator comments removed.

Test plan

  • Integration 262 (×2 runs), unit 857, smoke 3/3, editor-prepare task — all green locally
  • 3-agent pre-PR review (correctness / style / tests): no blockers; both minor findings fixed (stale tests/replay.py paths in .claude docs; stress.py silently inheriting the 1-worker test default)
  • Collection count identical to main (262); reviewers verified merges symbol-by-symbol and that no assertion was weakened

Summary by CodeRabbit

  • Documentation

    • Updated smoke-test and editor setup instructions to use the current tool locations in English and Chinese documentation.
    • Corrected usage examples for test preparation, replay, and stress testing.
  • Tests

    • Consolidated shared testing utilities to improve consistency across integration and stress tests.
    • Strengthened server lifecycle checks, shutdown handling, indexing waits, recompilation checks, and reference polling.
    • Standardized test initialization, workspace handling, and compile-command generation.

All framework/runner code moves out of the suite trees into one flat
package:

- Standalone scripts: replay.py (smoke), prepare.py (editor E2E),
  stress.py (manual), plus compile_commands.py holding every
  compile_commands.json writer (cmake generation, static test data,
  per-test entries). These stay stdlib-only so the editor pixi env
  can run prepare.py without pygls.
- Test library (was integration/utils): client.py (CliceClient),
  lifecycle.py (spawn/shutdown/clean-exit gate/ports), checks.py
  (diagnostics + anomaly + wait helpers), workspace.py (on-disk
  sources, document edits, cache inspection), injection.py.
- conftest shrinks to fixtures and hooks; the cache_dir/worker-count
  test policy lives once in CliceClient.initialize — agentic tests
  that initialize directly now get the same 3-process default as
  fixture-based ones.
- Resolves the tests/stress.py vs tests/integration/stress/ clash.
@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The PR consolidates integration-test helpers under tests/tools, centralizes lifecycle and fixture behavior, rewires integration-test imports, and updates documentation, commands, editor setup, and Pixi tasks to use relocated test scripts.

Changes

Test utility consolidation

Layer / File(s) Summary
Shared test helper implementations
tests/tools/checks.py, tests/tools/workspace.py, tests/tools/compile_commands.py
Adds shared workspace, compile-command, diagnostic-wait, indexing, and reference helpers.
Lifecycle and fixture wiring
tests/tools/lifecycle.py, tests/tools/client.py, tests/conftest.py
Centralizes port selection, client startup, shutdown validation, anomaly checks, cache configuration, and fixture initialization.
Integration test helper migration
tests/integration/**
Updates integration tests to import helpers from tests.tools instead of legacy utility modules and adjusts selected initialization options.
Test script and command paths
.claude/*, docs/*, pixi.toml, editors/vscode/*, tests/tools/{prepare,replay,stress}.py
Updates test commands, documentation, repository-root resolution, and standalone tool imports for the relocated scripts.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

  • clice-io/clice#409: Also restructures client lifecycle and shutdown helpers around tests/conftest.py.
  • clice-io/clice#456: Also changes fixture teardown and anomaly-check handling.
  • clice-io/clice#458: Adds editor test plumbing related to the updated editor-prepare paths.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 70.18% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the PR’s main change: consolidating shared test tooling under tests/tools.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch refactor/test-tools

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
tests/tools/compile_commands.py (1)

149-171: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider adding a std parameter to write_entries for parity with write_cdb.

write_cdb allows overriding the C++ standard, but write_entries hardcodes "-std=c++17". Minor inconsistency between the two sibling helpers; low priority since current callers only need c++17.

♻️ Optional parity fix
-def write_entries(workspace, entries):
+def write_entries(workspace, entries, *, std: str = "c++17"):
     """Write a compile_commands.json with per-file extra arguments.

     Args:
         workspace: Root directory of the workspace.
         entries: List of (file_name, extra_args) pairs; a file may appear
             multiple times to model multi-configuration projects.
+        std: C++ standard version (default: c++17).
     """
     data = [
         {
             "directory": str(workspace),
             "file": str(workspace / f),
             "arguments": [
                 "clang++",
-                "-std=c++17",
+                f"-std={std}",
                 "-fsyntax-only",
                 *args,
                 str(workspace / f),
             ],
         }
         for f, args in entries
     ]
     (workspace / "compile_commands.json").write_text(json.dumps(data))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/tools/compile_commands.py` around lines 149 - 171, Add an optional std
parameter to write_entries, defaulting to "c++17", and build the compiler
argument from it instead of hardcoding "-std=c++17". Keep the existing behavior
for current callers and align the parameter naming and handling with the sibling
write_cdb helper.
tests/tools/lifecycle.py (1)

121-145: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Log swallowed exceptions during best-effort shutdown.

Static analysis flags the three try/except Exception: pass blocks in shutdown_client. The best-effort teardown intent is reasonable (a crashed/already-dead server shouldn't blow up teardown), but silently discarding every exception makes flaky CI teardown failures hard to diagnose.

🔍 Suggested logging on swallowed exceptions
 async def shutdown_client(c: CliceClient, *, verbose: bool = False) -> None:
     """Gracefully shut down a client, force-kill if needed."""
     try:
         await asyncio.wait_for(c.shutdown_async(None), timeout=10.0)
-    except Exception:
-        pass
+    except Exception as exc:
+        print(f"[shutdown_client] shutdown_async failed: {exc!r}", flush=True)

     try:
         c.exit(None)
-    except Exception:
-        pass
+    except Exception as exc:
+        print(f"[shutdown_client] exit() failed: {exc!r}", flush=True)
     ...
     try:
         await assert_server_exited_cleanly(c.server)
     finally:
         try:
             await c.stop_io()
             await asyncio.sleep(0.1)
-        except Exception:
-            pass
+        except Exception as exc:
+            print(f"[shutdown_client] stop_io failed: {exc!r}", flush=True)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/tools/lifecycle.py` around lines 121 - 145, Update shutdown_client to
log exceptions from each best-effort shutdown operation instead of silently
passing: shutdown_async, exit, and stop_io. Preserve the existing teardown flow
and exception swallowing, but emit concise diagnostic messages including the
caught exception and identify which operation failed.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/tools/compile_commands.py`:
- Around line 149-171: Add an optional std parameter to write_entries,
defaulting to "c++17", and build the compiler argument from it instead of
hardcoding "-std=c++17". Keep the existing behavior for current callers and
align the parameter naming and handling with the sibling write_cdb helper.

In `@tests/tools/lifecycle.py`:
- Around line 121-145: Update shutdown_client to log exceptions from each
best-effort shutdown operation instead of silently passing: shutdown_async,
exit, and stop_io. Preserve the existing teardown flow and exception swallowing,
but emit concise diagnostic messages including the caught exception and identify
which operation failed.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: e44b4aef-bdf6-4a00-b7c7-5ee0c69020f9

📥 Commits

Reviewing files that changed from the base of the PR and between 2bafe72 and 27b4aca.

📒 Files selected for processing (47)
  • .claude/CLAUDE.md
  • .claude/commands/test.md
  • docs/en/dev/test-and-debug.md
  • docs/zh/dev/test-and-debug.md
  • editors/vscode/.vscode-test.mjs
  • pixi.toml
  • tests/conftest.py
  • tests/integration/agentic/test_agentic.py
  • tests/integration/agentic/test_cli.py
  • tests/integration/compilation/test_header_pch.py
  • tests/integration/compilation/test_pch.py
  • tests/integration/compilation/test_persistent_cache.py
  • tests/integration/compilation/test_self_containment.py
  • tests/integration/compilation/test_staleness.py
  • tests/integration/extensions/test_context_switching.py
  • tests/integration/extensions/test_header_context.py
  • tests/integration/features/test_completion.py
  • tests/integration/features/test_file_tracker.py
  • tests/integration/features/test_formatting.py
  • tests/integration/features/test_guidance_diagnostics.py
  • tests/integration/features/test_header_reindex.py
  • tests/integration/features/test_inactive_regions.py
  • tests/integration/features/test_index.py
  • tests/integration/features/test_index_staleness.py
  • tests/integration/features/test_query_freshness.py
  • tests/integration/features/test_server.py
  • tests/integration/lifecycle/test_anomaly.py
  • tests/integration/lifecycle/test_config.py
  • tests/integration/lifecycle/test_file_operation.py
  • tests/integration/lifecycle/test_protocol_edges.py
  • tests/integration/lifecycle/test_protocol_robustness.py
  • tests/integration/modules/test_modules.py
  • tests/integration/stress/test_eviction.py
  • tests/integration/stress/test_rapid_edit.py
  • tests/integration/utils/__init__.py
  • tests/integration/utils/wait.py
  • tests/integration/utils/workspace.py
  • tests/tools/__init__.py
  • tests/tools/checks.py
  • tests/tools/client.py
  • tests/tools/compile_commands.py
  • tests/tools/injection.py
  • tests/tools/lifecycle.py
  • tests/tools/prepare.py
  • tests/tools/replay.py
  • tests/tools/stress.py
  • tests/tools/workspace.py
💤 Files with no reviewable changes (3)
  • tests/integration/utils/workspace.py
  • tests/integration/utils/init.py
  • tests/integration/utils/wait.py

@16bit-ykiko
16bit-ykiko merged commit 83719e5 into main Jul 10, 2026
21 checks passed
@16bit-ykiko
16bit-ykiko deleted the refactor/test-tools branch July 10, 2026 06:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant